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Abstract 



The ATLAS experiment sensitivity to top quark Flavour Changing Neutral Cur- 
rent (FCNC) decays was studied at LHC using ti events. While one of the top 
quarks is expected to follow the dominant Standard Model decay t — > bW, the 
other decays through a FCNC channel, i.e. t — > Zu(c), t — > 711(c) or t — > gu(c). 
Different types of analyses, applied to each FCNC decay mode, were compared. 
The FCNC branching ratio sensitivity (assuming a 5a signal significance) and 
95% confidence level limits on the branching ratios (in the hypothesis of signal 
absence) were obtained. 
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1 Introduction 



Flavour Changing Neutral Currents (FCNC) are strongly suppressed in the Standard 
Model (SM) due to the Glashow-Iliopoulos-Maiani (GIM) mechanism [1]. Although absent 
at tree level, small FCNC contributions are expected at one loop level, according to the 
Cabibbo-Kobayashi-Maskawa (CKM) mixing matrix [2]. In the top quark sector of the 
SM, these contributions limit the FCNC decay branching ratios to the gauge bosons, 
BR t ^ q x {X = Z,~/,g), to below 10~ 10 . There are however extensions of the SM, like 
supersymmetry (SUSY) [3], multi-Higgs doublet models [4] and SM extensions with exotic 
(vector-like) quarks [5] , which predict the presence of FCNC contributions already at tree 
level and significantly enhance the FCNC decay branching ratios compared to the SM 
predictions. 

Due to its large mass, much higher than any other known fermion, the top quark is 
a very good laboratory to look for physics beyond the SM. If the top quark has FCNC 
anomalous couplings to the gauge bosons, its decay properties would be affected, and 
possibly measured at colliders, in addition to the dominant decay mode t — > bW. Indeed 
one of the most prominent signatures of FCNC processes at the Large Hadron Collider 
(LHC), would be the direct observation of a top quark decaying into a charm or an up 
quark together with a 7, g or Z boson [6]. In the effective Lagrangian approach [7,8] the 
new top quark decay rates to the gauge bosons [9], 

T{t^ qg )=^fj S -a s ml (1) 

/ 7 \ 2 

r(t - qi) = [^-J 2aml (2) 

T(t -> qZ)-, = (\vf s \ 2 + \aff) °'"? 4M | Bin 2 20w (l-^j (!+ 2 4)-d (3) 

can be expressed in terms of the «f g , K] q , (|f t g| 2 + |af,| 2 ) and K,f q anomalous couplings to 
the g, 7 and Z bosons respectively. The energy scale associated with this new physics 
is represented by A, while a s and a are, respectively, the strong and electromagnetic 
coupling constants. The electroweak mixing angle is represented by 9w and the top and 
Z masses are represented, respectively, by rrit and mz- 

Although FCNC processes associated with the production [10,11] and decay [12] of top 
quarks have been studied at colliders (BR t ^ Zq < 33% and BR t ^ iq < 3.2% at 95% Con- 
fidence Level (CL) [12]), the amount of top quark relevant data collected up to now is 
not comparable with the statistics expected at the LHC. The LHC will operate with a 
centre-of-mass energy of 14 TeV, and in the low luminosity phase (/ = 10 33 cm~ 2 s _1 ), 
several millions of top quarks will be produced per year and experiment, mainly in pairs 
(with a NLO cross-section of 833 pb [9,13]), but also through single top production (with 
an expected NLO cross-section of 280 pb [14]). 

This paper is devoted to the study of the ATLAS experiment [15] sensitivity to FCNC 
top quark decays at the LHC. While one of the top quarks is expected to follow the 
dominant SM decay (t — > bW), the other decays through a FCNC channel, i.e. t — > Zq, 
t 1Q or t — > gq. The corresponding Feynman diagrams are shown in Fig. [TJ Different 
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types of analyses (cut-based and likelihood-based) were applied to each FCNC decay mode 
and their results compared. 

This paper is organised as follows. After the introduction, a description of the simulated 
signal and background is given in section [2j The analysis criteria applied to each FCNC 
channel are described in section [3] and in section [4] a comparison of the results obtained 
by the different analyses [16,17] is presented within two different approaches: branching 
ratio sensitivities (assuming a 5a signal significance for discovery) and 95% confidence level 
limits (in the hypothesis of signal absence). These results are compared with previously 
published ones [10-12,15,18,19]. In section \5\ the conclusions are presented. 

2 Signal and background simulation 

The Monte Carlo (MC) generation of the QCD (66), W+jets, Z/ 7 *+jets, WW, ZZ and 
ZW background processes was done with the library PYTHIA [20]. Single top quark 
production was generated with TopReX 4.05 [21], and the SM top pair production (Usm) 
was generated using TopReX and PYTHIA. These libraries were also used to generate 
signal tt samples, where one of the t-quarks decays via Charged Currents (CC) into bW 
and the other one decays through FCNC into qZ, or qg. For TopReX the anomalous 
couplings to the g, 7 and Z bosons were set to n 9 tq = nj q = (|f£| 2 +|a£| 2 ) 1 ' 2 = = 0.1 and 
A was set to 1 TeV. The top mass was set to 175 GeV/c 2 . Different values, 170 GeV/c 2 and 
180 GeV/c 2 , were also considered for the study of systematic uncertainties, as explained in 
section I4T31 No SUSY backgrounds or other contributions beyond the SM were considered 
in the present analyses. The CTEQ2L and CTEQ5L Parton Distribution Functions (PDF) 
were used [20,21] in the analyses and the CTEQ4M was used for systematic studies. No 
pile-up was taken into account. 

The generated background and signal events were passed through the ATLAS fast 
simulation packages ATLFAST [22] and ATLFASTB [22]. For each event, these packages 
begin by simulating the energy deposition in the calorimeter cells of all the stable parti- 
cles. The calorimeter cells are clustered within a cone of AR = v/(A0) 2 + (At/) 2 = 0.4. 
Cells with Et > 1.5 GeV are used as cluster seeds and the cone algorithm is applied in 
decreasing order of Et- Only clusters with Et > 5 GeV are considered. The polar angle 
and the momentum of photons are smeared according to Gaussian parameterizations. For 
electrons, their momenta are smeared according to a Gaussian parameterizations. The 
momentum of each muon is smeared according to a resolution which depends on the p?, \t)\ 
and 0. The photon (electron) energy resolution is 6E/E < 2.9% (3.3%), for E > 20 GeV. 
The transverse momentum resolution of muons with pt < 100 GeV/c is Spt/pt ^ 2%. 
Photons, electrons and muons are selected only if they have \r)\ < 2.5 and pt > 5 GeV/c 
(Pt > 6 GeV/c for muons). They are classified as isolated if the transverse energy of the 
cluster associated to the particle, inside a cone of AR = 0.2, does not exceed 10 GeV the 
particle energy, and the AR from other energy clusters must be above 0.4. The clusters 
of energy depositions not associated to isolated photon, electrons or muons are used for 
the jet reconstruction. Their momenta are smeared according to a Gaussian distribution 
which depends on \rj\. Jets are selected if they have E T > 10 GeV. For E > 20 GeV, 
the jet energy resolution is less than 12% (\r}\ < 3) and less than 24% (\r}\ > 3). The 
missing transverse momentum is estimated by summing the transverse momentum of the 
isolated photons, electrons, muons and jets. The non isolated muons and the clusters of 
energy deposition not associated to isolated photons, electrons, muons or jets, are also 
taken into account. In the ATLAS detector, it will be possible to identify 6-jets with 
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\rj\ < 2.5 by using 6-tagging tools. The algorithm was simulated by setting a 6-tagging 
efficiency to 60%, with contamination factors set to 14.9% (1.1%) for c-jets (light quark, 
gluon and tau jets) for the low luminosity phase (I = 10 33 cm _2 s _1 ). In order to check the 
dependence of the analysis with the 6-tagging efficiencies, different values, 50% and 70% 
(corresponding to the expected 6-tag variation within the interesting signal transverse mo- 
mentum range), were also considered for the systematic studies and the high luminosity 
phase (/ = 10 34 cm^s" 1 ), with contamination factors of 9.2% (0.4%) and 23.3% (2.9%) 
for c-jets (light quark, gluon and tau jets), respectively. 

Initial and final state QED and QCD radiation (ISR+FSR), multiple interactions and 
hadronization were taken into account in the event generation. Due to the hadronization 
and FSR, the jets are reconstructed with less energies than those from the original quarks 
or gluons. The jets energies were calibrated by the ATLFASTB package, by applying a 
calibration factor, K^ et = p^ vton / pl^" , that is the ratio between the true parton energy and 
the reconstructed jet energy. The calibration factor depends on the pt and is different for 
fe-tagged and light jets. 

Preliminary full simulation studies, based on the ATHENA framework [23] , indicate a 
fair agreement between the fast and full simulations of the ATLAS detector. 

3 Topologies and event selection 

The ti final states corresponding to the different FCNC top decay modes lead to different 
topologies according to the number of jets, leptons and photons. There is however a 
common characteristic of all channels under study, i.e. in all of them one of the top 
quarks is assumed to decay through the dominant SM decay mode t — > bW and the other 
is forced to decay via one of the FCNC modes t — > Zq, t — > ^q or t — > gq. Two different 
types of analyses, labelled "cut-based" and "likelihood-based", were used to study the 
ATLAS sensitivity to FCNC top quark decays. For both analyses the leptonic decays of 
the W (W — ► ivi) were taken into account^. In addition, for the FCNC channel t — > Zq, 
the hadronic decay of the W (W — ► qq') was also considered for the cut-based analysis. 

3.1 t — > Zq channel 

The QCD backgrounds at hadron colliders make the search for the signal via the fully 
hadronic channel (when both the W and Z decay hadronically) very difficult. For this 
reason only the leptonic decay of the Z was considered. The final state was then deter- 
mined by the decays of the W boson. Two different possible decay channels have been 
considered: the first ('leptonic mode') where the W decays leptonically W — > Iv, and the 
second ('hadronic mode') with W — > jj. The hadronic W decay signature has a larger 
branching fraction, but suffers from larger backgrounds. The experimental signature of 
the leptonic mode includes three isolated charged leptons, two of which reconstruct a Z 
boson, large missing transverse energy due to the neutrino and at least two jets, one of 
which is tagged as a 6-jet. The signature of the hadronic mode is characterised by having 
two leptons (again with rri£+£~ « mz) and at least four jets, one of which is tagged as 
a 6-jet. Following a previous analysis [18], new cut-based [16] and likelihood-based [17] 
analyses were developed and are described below. 

1 For the cut-based analyses i = e,fi, while for the likelihood-based analyses i = e,/i,T 
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3.1.1 Cut-based analysis: hadronic mode 

The final state for the hadronic W decay mode is ti — > ZqWb — > £ + £~ jjjb. This mode has 
the following backgrounds: Z+jets production, followed by the decay Z — > £ + £~, pp — > 
W ± Z + X -> jj£ + £~ + X, and it -> with the final state topologies (a) £ + vb£~vb, 

or (b) £±vbjjb. In the case of (a), the additional two jets must come from QCD radiation, 
while in (b) the source of leptons is from cascade decays. Z+jets production at the LHC 
has a relatively large cross-section, dominated by qg — > Zq and gg — > Zg processes. To 
decrease the size of the background sample which needed to be generated, thresholds were 
imposed at the generator level on the invariant mass, m = \f§ > 130 GeV/c 2 , where a/§ is 
the effective centre-of-mass energy, and transverse momentum, p? > 50 GeV/c, of the hard 
scattering process. The cross-section for this subsample of events was crz+jets=3186 pb. 
The WZ background is the electroweak process pp — > W^Z + X, and has an assumed 
cross-section of owz = 26.58 pb. Background samples of 2.1 x 10 7 Z+jets events, 1.2 x 10 5 
WZ events and 2.8 x 10 7 ti events were generated. Assuming the above mentioned 
production cross-sections, and including the relevant branching ratios, these background 
samples correspond to an integrated luminosity of 100 fb _1 . 

The analysis began with preselection cuts requiring that the event contains at least two 
charged leptons (electrons with p T > 5 GeV/c within pseudorapidity \r)\ < 2.5 and muons 
with pt > 6 GeV/c within pseudorapidity \tj\ < 2.4), and include a pair of opposite-sign 
and same-flavour leptons, compatible with them having come from a Z decay. In addition, 
the number of jets with PTj C t > 15 GeV/c within pseudorapidity \r)\ < 5.0 was required 
to be at least four. After preselection cuts, 46% of the signal events were accepted, while 
only 3.0%, 3.5% and 4.1% of the ti, Z+jets and WZ background events, respectively, 
were selected. 

The next cuts required the presence of two isolated leptons with p Te > 20 GeV/c and 
the demand for at least four jets with PTjet > 50 GeV/c and \r]^\ < 2.5. The isolation 
AR of the jets (measured in relation to other jets and leptons) was then required to be 
greater than 0.4. 

Fig. [2] presents the distributions of reconstructed dilepton invariant mass and of recon- 
structed t — > Zq invariant mass rriuj for the best combinations of ££j (i.e. ££j combination 
with the closest to top mass value of invariant mass m^j) for the signal sample. A cut 
was then applied on the dilepton invariant mass, requiring that it lie within ±6 GeV/c 2 
around m>z- 

To suppress the large remaining Z+jets background, it was necessary to use the in- 
formation that signal events contain, in addition to the decay t — > Zq, a hadronic decay 
t — > Wb — > jjb of the other top quark. The hadronic top quark decay was, therefore, 
reconstructed as part of the signal requirement. First, a pair of jets was required to have 
an invariant mass rrijj within a 16 GeV/c 2 around mw Fig. [3] shows the distribution of 
reconstructed rrijj for the best combinations of jj (i.e. jj combination with the closest 
to W mass value of invariant mass rrijj) for the signal events. The W mass resolution is 
(7 m ,. = 8 GeV/c 2 . A requirement was then made to have exactly one jet tagged as a 6-jet. 
Finallythe jjb invariant mass was required to lie within 8 GeV/c 2 around m t . 

Fig. [3] presents the distribution of the reconstructed invariant top mass {rrijjb) for the 
best combinations of jjb (i.e. jjb combination with the closest to top mass value of 
invariant mass rrijjb) fo r th e signal. The top mass resolution is <j(rrijjb) = 18.5 GeV/c 2 , 
implying that the mass window applied is rather narrow in order to get a large background 
rejection. The sequence of cuts required to reconstruct the hadronic decay of the other 
top quark dramatically suppresses the backgrounds, but also reduces the signal efficiency 
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TaWe 1: The number of events, normalised to L = 100 fb , and efficiency (%) of selection cuts applied 
in sequence for the signal and backgrounds in the hadronic decay mode in the t — > Zq channel, obtained 
with a cut-based analysis, are shown. 

by almost an order of magnitude. For those events with an accepted t — > jjb candidate, 
the invariant mass of the Z candidate with the remaining unassigned high jets was 
reconstructed to look for a signal from t —>■ Zq decays. The resolution a of muj distri- 
bution is a mu . = 9.9 GeV/c 2 (see Fig. [2]). The analysis cuts reduce the WZ background 
to a negligible level in the mz q ± 24 GeV/c 2 mass window. Two events of the Z+jets 
background are accepted in this mass window. 

Table [1] summarises the effects of the sequential application of the above described 
various analysis cuts on the background samples and on the sample of 19000 signal events 
of the topology tt — > ZqWb — > £ + £~jjjb. 

3.1.2 Cut-based analysis: leptonic mode 

The final state for the leptonic decay mode is tt — > ZqWb — > £ + £~ j£vb. The experimental 
signature therefore includes three isolated charged leptons, two of which reconstruct a Z 
boson, and large missing transverse energy due to the undetected neutrino. 

This mode has the following backgrounds: Z(— > ££)+jets, pp — > W^Z+X — > £ ± v£ + £~ + 
X, and tt — > W + bW~b — > £ + vb£~vb. Assuming the production cross-sections given ear- 
lier, and including the relevant branching ratios, background samples of 2.1 x 10 7 Z+jets 
events, 38000 WZ events, and 3.9 x 10 6 tt events were generated. These background 
samples correspond to an integrated luminosity of 100 fb _1 . 

Preselection cuts were first applied, requiring the presence of at least three charged 
leptons (electrons with pr > 5 GeV/c and muons with pt > 6 GeV/c) within pseudora- 
pidity < 2.5. Of these, at least one pair of leptons must be of opposite sign and same 
flavour, compatible with them being produced from a Z decay. In addition, the number 
of jets in the event with Pr^t > 15 GeV/c within pseudorapidity \r]\ < 5.0 was required 
to be at least two. The requirement of three leptons reduces significantly the Z+jets and 
ti backgrounds, while the requirement of two jets reduces significantly WZ and Z+jets 
backgrounds. 

The lepton criteria were then tightened, by requiring the presence of at least three iso- 
lated, charged leptons (electrons or muons) with pri > 20 GeV/c. The next requirement, 
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Table 2: The number of events, normalised to L = 100 fb~ x , and efficiency (%) of selection cuts applied 
in sequence for the signal and backgrounds for the leptonic mode in the t — » Zq channel, obtained with a 
cut-based analysis, are shown. 

namely that the missing transverse momentum in the event satisfies $t > 30 GeV/c, is 
effective at further reducing the Z+ jets background while having little impact on the 
signal and other background sources. Next, it was demanded that there be at least two 
jets with Pt jet > 50 GeV/c, |?7j C t| < 2.5, and satisfying the following isolation conditions: 
ARjj > 0.4 (jet-jet isolation) and ARij > 0.4 (lepton-jet isolation). The cut requiring 
the presence of two or more jets in each event effectively suppresses the WZ background. 

The presence of a reconstructed Z — ► £ + £~ decay is a powerful cut against the tt 
background. A like-sign, same-flavor pair of isolated leptons was required to reconstruct 
to the Z mass within rriz ± 6 GeV/c 2 . Fig. H] presents the distribution of reconstructed 
invariant mass of ££ pairs m«, for all dilepton combinations for the signal events. The 
width of the accepted window corresponds to approximately twice the Z mass resolution 
of about 2.9 GeV/c 2 . The next requirement was the presence in the event of exactly one 
tagged 6-jet, which is effective at further reducing the WZ background Finally, a peak 
at the top quark mass in the Zj invariant mass distribution was sought. In Fig. HJ the 
distribution of reconstructed invariant mass rnuj f° r an combinations of ££j is presented 
for the signal events. The top quark mass resolution is a(muj) = 14 GeV/c 2 . Accepted 
combinations were required to lie within ±24 GeV/ c 2 (~ 2a) around the known top quark 
mass. This top mass window removes almost completely the remaining background and 
the final signal efficiency is 6.1% with a total background of 7 events for an integrated 
luminosity of 100 fb _1 . 

Table [2] summarises the effects of the sequential application of the above described 
various analysis cuts on the background samples and on the sample of 20565 signal events 
of the topology tt -> ZqWb £ + £~]£vb. 

3.1.3 Likelihood-based analysis: leptonic mode 

The leptonic decay mode was also studied with a likelihood-based analysis. A general 
selection criteria was defined for the likelihood-based analyses: events were required to 
have at least one isolated lepton with p T > 25 GeV/c and at least two jets with p T > 
20 GeV/c in the final state. Both the lepton and the jets were required to have \rj\ < 2.5. 
Additionally, the transverse missing momentum had to be greater than 20 GeV/c. Table [3] 
summarises the cuts performed in the likelihood-based analysis. 

At the preselection, events were accepted if they had at least two additional isolated 
leptons (apart from the one already required by the general selection criteria) with px > 
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Ta6Ze 5: Selection cuts applied to the likelihood-based analyses. 



10 GeV/c and |?7| < 2.5. For the likelihood-based analyses, all the background samples 
were normalised to L = 10 fb _1 . The number of selected background events and the 
signal efficiency are shown in Table |U The distribution of relevant variables at this level 
are shown in Fig. [51 

The final event selection was done by requiring the leading jet to have pt > 30 GeV/c. 
One of the jets had to be tagged as a 6-jet. Additionally, in order to be compatible with 
the Z — > decay, two of the three leptons present in the final state were required to 
have opposite charges and the same flavour (electron or muon). The number of selected 
SM events and the signal efficiency at the final selection level are shown in Table HI The 
dominant contribution for the single top background is the t-channel. No QCD (bb) and 
W+jets events passed the final selection criteria, within the generated statistics (3.75 x 10 8 
and 3.5 x 10 7 events, respectively). 

The reconstruction of the Z mass was done by calculating the £ + £~ invariant mas^l 
(mi+£-) and is shown in Fig. [6^. The reconstruction of the mass of the t-quark which 
decayed through FCNC (rriji+i-) was done by associating the non-6 jet with the £ + £~ 
pair. If more than one non-6 jet was present, the one with highest p? was chosen. The 
signal and SM distributions of w,ji+i- are shown in Fig. [Hb. The decay of the other t- 
quark (t — > btv) cannot be directly reconstructed due to the presence of an undetected 
neutrino in the final state. Nevertheless, the neutrino four-momentum can be estimated 
by assuming the transverse missing energy to be the transverse neutrino momentum. The 
longitudinal component can be determined, with a quadratic ambiguity, by constraining 
the W mass (calculated as the invariant mass of the neutrino and the most energetic 
remaining lepton) to its known central value {my/ = 80.4 GeV/c 2 [24]). The mass of t- 
quark with a SM decay, shown in Fig. Eb, was reconstructed by associating the 6-jet with 
the reconstructed W. The quadratic ambiguity was removed by choosing the solution 
closer to m t = 175 GeV/c 2 . 

Following the final selection, a likelihood-based type of analysis was applied. Sig- 
nal (j?p 9na ^ anc } background-like ("P^ acfc ) probabilities were computed using Probabil- 
ity Density Functions (p.d.f.), constructed from relevant physical variables. The signal 
Cs = H™ =1 'pp 9nal and background £b = W£ = {P\ ack " likelihoods (n is the number of p.d.f.) 
were used to built the discriminant variable, defined as L R = In (Cs/Cb), assuming un- 
corrected variables. 

2 In the case where more than one pair of leptons had the same flavour and opposite charges, the pair with highest px 
was chosen. 
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Table 4- The number of selected background events, normalised to L = 10 fb , and signal efficiencies in 
the t — > Zq channel for the preselection and final selection levels, obtained with a likelihood-based analysis, 
are shown. 

For the t — > Zq channel the p.d.f. were based on the following physical distributions 
(c.f. Fig. ED: 

• minimum invariant mass (m^) of the three possible combinations of two leptons 
(only the three leading leptons were considered); 

• transverse momentum of the third lepton (ptis)] 

• the j£ + £~ invariant mass and 

• the transverse momentum of the most energetic non-6 jet (pr,)- 

The Lr distributions for SM background and signal are shown, after the final selection, 
in Fig. [SI The number of expected SM background as a function of the signal efficiency 
obtained by cutting the discriminant variable is shown in Fig. [91 

3.2 t — ► 'yq channel 

The tt final states corresponding to the FCNC decay t — ► are characterised by the 
presence of a high pt photon and a light jet from the top quark decay. Since the existence 
of the photon is not sufficient to reduce the QCD background, only the leptonic decays of 
the W (originated from the SM decay of the other top quark) were considered. The final 
states corresponding to these signal events are characterised by a topology with two jets 
(one 6-jet from the SM top decay), one high p? photon, one lepton and missing transverse 
momentum from the undetected neutrino. Following a previous analysis [18], a new one 
was developed [17] and is described below. 

After applying the general selection criteria described in section 13.1.31 a preselection 
was defined by requiring the events to have at least one photon with px > 50 GeV/c and 
\tj\ < 2.5. Additionally, in order to prevent events to be simultaneously assigned to the 
t — ► Zq and t — ► jq channels, a maximum of two leptons in the final state were allowed. 
The number of selected SM background events and the signal efficiency at the preselection 
level are shown in Table The distributions of relevant variables for SM background and 
signal are shown in Fig. [101 

The final event selection was done by requiring the leading photon to have pt > 
75 GeV/c and one of the two jets with highest pr to be tagged as a 6-jet (only one 6-jet 
was allowed). This selection largely reduces the background, being tisM the dominant 
contribution, as can be seen in Table [5j The FCNC top decay was reconstructed using 
the non-6 jet and the photon (in the cases were more than one photon or non-6 quark 
were available, the one with higher px was chosen). No QCD (66) and VT+jets events, 
within the generated statistics, passed the final selection criteria. Just like for the t — > Zq 
channel, a likelihood-based type of analysis was used. The p.d.f. were built based on the 
following variables (c.f. Fig. [TT]) : 



• the mass of the t-quark with FCNC decay, reconstructed from the photon and the 
non-6 jet (TOj 7 ); 

• the transverse momentum of the leading photon (pr 7 ) an d 

• the number of jets. 

The discriminant variables distributions for signal and SM expectation are shown in 
Fig. [T2] and the number of expected SM background as a function of the signal efficiency 
obtained by cutting the discriminant variable is shown in Fig. [13] 

3.3 t — ► gq channel 

The final states of tt events with one of the top quarks decaying into a gluon, t — *> gq, are 
characterised by the presence of a high gluon and a light jet from the top quark decay. 
Only the leptonic decays of the W (originated from the SM decay of the other top quark) 
were taken into account, otherwise the final state would be fully hadronic and the signal 
would be overwhelmed by the QCD background. The final states are thus characterised 
by the existence of at least three jets (one 6-jet from the SM top decay), one lepton and 
missing transverse momentum from the undetected neutrino. 

Although no previous analyses have been performed for the t — > gq decay, the anoma- 
lous coupling tgq in top production was studied in the past [15,19]. A new analysis 
dedicated to the t — ► gq decay was developed [17] and is described here. 

As in this topology the FCNC top decay corresponds to a fully hadronic final state, 
a more restrictive event selection was necessary. The general selection criteria of sec- 
tion 13.1.31 was applied to the events. At the preselection, events were required to have 
only one lepton and no photons with transverse momentum above pt > 5 GeV/c, to reject 
events assigned to the other FCNC channels. The total visible energy (E vis ) had to be 
greater than 300 GeV. At least three jets with < 2.5 and pr > 20 GeV/ c were required. 
For the leading jet the cut was increased to 40 GeV/c. The events were then classified as 
"3 jets" or "4 jets" if they had exactly three jets or at least 4 jets, respectively. 

3.3.1 The "3 jets" sample 

The preselection was completed by requiring only one 6-tagged jet in the event. The 
gluon jet was assumed to be the non-6 jet with the highest transverse momentum. This 
distribution is shown in Fig. [T3J together with the mass of the t-quark with FCNC decay 
{m q g), reconstructed from the non-6 jets. The mass of the t-quark with SM decay (m&&,, 
reconstructed according to section 13.1.31) is also shown. The number of selected SM 
background events and the signal efficiency at this level are presented in Table [6] The 



Description 
of 
Cuts 


Signal 

t -> -yq 


Background Processes 


Z+jets 


Z + W 


tt 


single t 


e(%) 


Nevt 


Nevt 


Nevt 


Nevt 


Preselection 


23.3 


584.2 


325.7 


2832.4 


206.2 


Final Selection 


6.9 


15.2 


7.7 


271.6 


23.0 



Table 5: The number of selected background events, normalised to L = 10 fb , and signal efficiencies in 
the t — ► 7<7 channel for the preselection and final selection levels, obtained with a likelihood-based analysis, 
are shown. 
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Description 
of 
Cuts 


Signal 
t -> gq 


Background Processes 


Z+jets 


Z + W 


tt 


single t 


VF+jets 


e{%) 


Nevt 


Nevt 


Nevt 


Nevt 


Nevt 


Preselection 


1.6 


1356.6 


427.1 


24366.7 


11328.2 


23320.3 


Final Selection 


1.2 


157.1 


22.1 


4985.6 


1187.9 


1813.3 



Table 6: The number of selected background events, normalised to L = 10 fb , and signal efficiencies in 
the t —> gq channel ("3 jets") for the preselection and final selection levels, obtained with a likelihood-based 
analysis, are shown. 



Description 
of 
Cuts 


Signal 
t gq 


Background Processes 


Z+jets 


Z + W 


tt 


single t 


IF+jets 


e(%) 


Nevt 


Nevt 


Nevt 


Nevt 


Nevt 


Preselection 


5.7 


1171.0 


305.2 


216679.9 


14263.1 


12651.2 


Final Selection 


1.2 


64.3 


7.1 


9142.1 


453.3 


379.5 



Table 7: The number of selected background events, normalised to L — 10 fb , and signal efficiencies in 
the t — > gq channel ("4 jets") for the preselection and final selection levels, obtained with a likelihood-based 
analysis, are shown. 

final event selection was done by requiring the gluon to have pt > 75 GeV/c and 125 < 
m qg < 200 GeV/c 2 . No generated QCD (66) events passed the final selection criteria. 

As for the other channels, a likelihood-based type of analysis was adopted, using the 
following variables to build the p.d.f. (c.f. Fig. [TBI) : 

• the qg invariant mass, 

• the blv invariant mass, 

• transverse momentum of the 6-jet (pt&), 

• transverse momentum of the second non-6 jet (pr,) an d 

• angle between the lepton and the gluon (c%). 

The signal and background discriminant variable distributions are shown in Fig. [161 
The number of expected SM background as a function of the signal efficiency obtained 
by cutting the discriminant variable is shown in Fig. [T7J 

3.3.2 The "4 jets" sample 

For this topology, the preselection was completed by requiring the fourth jet to have 
Pt > 20 GeV/c and \r]\ < 2.5. Only one 6-tagged jet, which had to be among the first 
four, was allowed in the event. The gluon jet was assumed to be the non-6 jet with the 
highest transverse momentum. This distribution is shown in Fig. [TSl together with the 
mass of the t-quark with FCNC decay (m g j), reconstructed from the two non-6 jets with 
the highest transverse momenta. The mass of the t-quark with SM decay is also shown. 
The number of selected SM background events and the signal efficiency at this level are 
presented in Table [71 

The final selection was defined by requiring the gluon transverse momentum to be 
above 100 GeV/c and the reconstructed mass of the t-quark with FCNC decay above 
150 GeV/c 2 and below 190 GeV/c 2 . As for the "3 jets" channel, no generated QCD (66) 
passed the final selection criteria. 
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After the final selection, the p.d.f. were built based on the following physical distribu- 
tions (c.f. Fig.dHD: 

• minimum invariant mass of the leading and the second non-6 jets or the leading and 
the third non-6 jets (m g j), 

• the blv invariant mass, 

• transverse momentum of qg, 

• transverse momentum of biu, 

• angle between the lepton and the gluon (a^ 9 ), 

• angle between the lepton and the 6-jet (c^&) and 

• angle between the gluon and the second non-6 jet (a gq ). 

The discriminant variable distributions for signal and SM expectation are shown in 
Fig. [2(3, while the number of expected SM background as a function of the signal efficiency 
obtained by cutting the discriminant variable is shown in Fig. EJJ 



4 Results and systematic studies 

Expected top quark FCNC decay branching ratios sensitivities of the ATLAS experiment 
were estimated for both the cut-based and likelihood-based analysis under two different 
hypothesis, as explained in the next subsections. 



4.1 Branching ratio sensitivity (5<x significance discovery hypothesis) 

Assuming a signal discovery with a 5a significance, the branching ratio {BR) sensitivity 
for each channel studied is estimated by: 

BR = , (5) 

2 x L x aittsM) x £t x 

where <j{tt SM ) = 833 pb [13] is the NLO calculation of the SM cross-section for tt produc- 
tion in pp collisions at y/s = 14 TeV. B is the total number of selected background events, 
St is the signal efficiency convoluted with the appropriate branching ratios and = 0.9 n is 
the charged leptons identification efficiency (n is the number of leptons required for each 
channel). The factor 2 in the denominator takes into account the t and i contributions to 
the BR. 

To evaluate the expected branching ratio sensitivities for a 5a signal significance of 
discovery in the cut-based analyses, the kinematic cuts were applied in sequence for the 
signal and backgrounds. In the channels studied using likelihood-based analyses, the ex- 
pected branching ratio sensitivities were evaluated after applying cuts to the discriminant 
variables, as given in Table M (see also Fig. 191 [TBI fTTI and I2T1) . These cuts were optimised 
according to the best S/VB (S is the number of selected signal events). The expected 
branching ratio sensitivities for a 5a discovery are shown in Table M 
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channel 


ifl CUt 


B 


e t (% ) 


t -> Zq 


> 5.62 


0.50 


0.06 


t -> 79 


> 2.71 


3.48 


0.62 




"3 jets" 


> 1.13 


734.1 


0.20 


"4 jets" 


> -0.38 


4033.9 


0.29 



Table 8: The number of selected background events (normalised to L — 10 fb^ 1 ) and signal efficiencies 
(convoluted with the appropriate branching ratios) for each channel of the likelihood-based analyses after 
the specified Lr cut are shown. 



channel 


type 




BR (L = 10 ftr 1 ) 


BR (L = 100 ft.- 1 ) 


t -> Zq 


cut-based 


hadronic 


1.7 x 10~ a [18] 


5.0 x 10~ 4 


leptonic 


4.7 x 10~ 4 [18] 


1.1 x 10~ 4 


likelihood-based 


leptonic 


4.4 x 10~ 4 


1.4 x 10~ 4 


t -> 79 


cut-based 






1.0 x 10~ 4 [18] 


likelihood-based 




9.4 x 10"° 


3.0 x 10- D 


t -> gq 


likelihood-based 


"3 jets" 


4.3 x 10~ a 


1.4 x 10~ 3 


"4 jets" 


6.9 x 10~ a 


2.2 x 10~ a 



Table 9: The branching ratio sensitivity for each channel in the 5cr discovery hypothesis is shown. The 
results for a luminosity of L — 10 and 100 fb^ 1 are presented. The values shown for the likelihood-based 
analyses were obtained after applying the cuts described in tabled The values presented for the cut-based 
analyses of the t — > Zq channel, with L = 10 fb , and of the t — > 79 channel were taken from Ref. [18] 

4.2 95% confidence level limits (hypothesis of absence of signal) 

In the absence of a FCNC top decay signal, expected limits at 95% CL can be derived. 
These limits were obtained for both the cut-based and the likelihood-based analyses, 
setting the charged lepton identification efficiency to 90%. 

For the cut-based analyses of the t —>■ Zq channel, the 95% CL upper limits were eval- 
uated considering an integrated luminosity of 100 fb -1 . Assuming the Poisson processes 
with backgrounds, 95% CL upper limits on the number of signal events for both decay 
modes were derived. The modified frequentist likelihood method [25] was used to evaluate 
the 95% CL upper limits for the likelihood-based analyses. The full information of the 
discriminant variables were used to derive 95% CL upper limits on the number of signal 
events for each channel. No cuts on the discriminant variables were used. Using the NLO 
calculation for <j(ttsM)t these limits were then converted into limits on the branching 
ratio for each of the studied FCNC top decay channels. The expected 95% confidence 
level limits on the branching ratios are summarised in Table [TUJ 

4.3 Systematic uncertainties and analyses stability 

The effect of different systematic sources of uncertainty on the limits evaluation was 
studied for both the cut-based and the likelihood-based analyses. This estimation was 
done by considering the changes on the central values of the signal efficiency, number of 
background events and likelihood ratio distributions. 

For the cut-based analysis of the t — *> Zq channel several systematic uncertainties were 
studied. The effect of the mass window cut applied to the top quark (which decayed 
through the FCNC channel) was studied by selecting events in a more restrictive window 
i.e., mi±12 GeV/c 2 (~ a). This results in a increase 12% (8%) on the 95% CL limit for the 
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hadronic (leptonic) mode. Varying the px cut applied to the jets from 50 to 40 GeV/c 2 
(leptonic mode) results in a relative change of the limit of 23% [26]. This variation 
significantly increases the tt, the WZ and partly the Z+jets backgrounds. Changing the 
lepton isolation criteria (to AR = 0.2) gives a relative change of 1% in the 95% CL limit. 

For the likelihood-based analysis of the leptonic mode of t — > Zq, the t — > jq and 
the t — > gq channels, the following systematic uncertainties were considered. The effect 
of the top mass uncertainty was evaluated using different Monte Carlo samples with 
m t = 170 GeV/c 2 and m t = 180 GeV/c 2 . This systematic uncertainty affects both the 
event kinematics (and consequently the discriminant variables shape) and the value of 
the tt cross-section (used in the limits evaluation). The overall theoretical uncertainty 
on (j(ti) was estimated in reference [9]. The effect of this uncertainty was studied by 
allowing a change of 12% on the central value of cr(tt), cross-section used both in the Usm 
background normalisation and in the BR limits evaluation, assuming a negligible error 
on the measurement itself. If the error on the cross-section measurement is, for instance, 
5%, the ATLAS sensitivity will be degraded but the change will not affect the order of 
magnitude of the results shown in this paper. For the t — > Zq and the t — > 'jq channels, 
a 5% error gives a maximum change on the limit of 5%. For the t —>■ gq channels, where 
the expected number of background is more important, the limit can change by a factor 
2 to 3 (depending if it is the 3 or 4 jets topology). A precise measurement of the tt 
cross-section is, for this reason, of utmost importance. The CTEQ 5L PDF set was used 
in the Monte Carlo generation. A different PDF set (CTEQ 4M [20,21]) was used to 
estimate the effect of this choice on the event kinematics. As mentioned in section [2, 
the ATLFASTB package was used to simulate the 6-tag algorithm with a 6-tag efficiency 
of 60%. In order to study the impact of the algorithm with a different efficiency, the 
6-tagging efficiencies of 50% and 70% were also considered. This source of uncertainty 
affects the signal efficiency, background estimation and discriminant variable shapes. The 
impact of the knowledge of the absolute jet energy scale was estimated by recalibrating 
the reconstructed jet energy. A miscalibration of ±1% for light jets and ±3% for 6-jets 
was used. This uncertainty was found to have a negligible effect on the signal efficiency, 
background estimation and discriminant variable shapes. A jet energy miscalibration of 
±5% for all jets was also considered. For the t — > Zq, t — > 79 and t — > gq ("3 jets") 
channels the relative changes on the 95% CL expected limits were found to be below 7%. 
For the most difficult channel {t gq- "4 jets") this effect is more important (up to 12%), 
due to the tighter selection criteria used to reject the large contribution from background. 
The stability of the cut-based analysis was studied by changing the preselection and final 
selection (typically a ±10% variation on the cut values was considered). The discriminant 



channel 


type 




BR (L = 10 lb" 1 ) 


BR (L = 100 fb- 1 ) 


t -> Zq 


cut-based 


hadronic 




2.7 x 10~ 4 


leptonic 




6.3 x lQ- b 


combined 




5.5 x 10- a 


likelihood-based 


leptonic 


3.1 x 1CT 4 


6.1 x 10~° 


t -> jq 


likelihood-based 




4.1 x 1CT 5 


1.2 x 10~ 5 


t gq 


likelihood-based 


"3 jets" 


1.6 x lCT 3 


4.8 x 10~ 4 


"4 jets" 


2.4 x lCT 11 


7.5 x 10~ 4 


combined 


1.3 x Kr a 


4.2 x 10~ 4 



Table 10: The expected 95% confidence level limits on the FCNC top decays branching ratio in the absence 
of signal hypothesis are shown. The results for a luminosity of L = 10 and 100 fb^ 1 are presented. 
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Source 


t -> Zq 


t -> jq 


t -> gq 


(3 jets) 


(4 jets) 


LU J J.1J.CLOD 


18% 


13% 


8% 


7% 


a(tt) 


11% 


11% 


9% 


7% 


PDFs choice 


15% 


7% 


3% 


6% 


6-tag algorithm efficiency 


16% 


5% 


18% 


17% 


jet energy calibration 


2% 


1% 


2% 


3% 


analysis stability 


9% 


12% 


3% 


13% 


p.d.f.s choice 


10% 


15% 


1% 


2% 



Table 11: Absolute value of the maximum relative changes on the 95% confidence level expected limits for 
each FCNC top decay branching ratio evaluated with the likelihood-based analyses. The reference values 
were presented in Table [7TJI (L = 10 fb~ x ). 

variables were computed using the probability density function sets described in section 
In order to estimate the effect of a different p.d.f. set, the following changes were studied: 
in the t — > Zq channel, the t reconstruction was done by considering the jet closest to the 
reconstructed Z in the invariant mass evaluation. Similarly, the i mass reconstruction in 
the t —> jq channel was done using the jet closest to the leading 7. Moreover, the t mass 
was included in the p.d.f. set and the multiplicity of jets with \rj\ < 2.5 was chosen as 
p.d.f. (instead of the jet multiplicity). In the t — > gq channel, AR was used instead of 
the angles, in the p.d.f.s definition. 

The absolute value of the maximum relative effect on the 95% confidence level expected 
limits on each considered source of systematic uncertainty (the reference values are those 
presented in Table [TUT) is shown in Table [TT] (L = 10 fb _1 ). Although differences up to 
20% were observed (caused by the uncertainty of the top mass), the order of magnitude 
of the expected limits on the BR is not affected by any of the systematic uncertainties 
considered. Moreover, the change on the selection criteria and on the p.d.f. sets do not 
have a significant impact on the results. 

5 Conclusions 

The sensitivity of the ATLAS experiment to the FCNC t — > qX (X = Z, 7, g) decays 
of the top quark was estimated. Different types of analysis (cut-based and likelihood- 
based) were used to obtain the FCNC branching ratio sensitivities (assuming a 5a signal 
significance for discovery) or the 95% CL limits on the FCNC branching ratios (in the 
absence of signal). The leptonic mode of the t — > Zq channel was studied with both 
type of analysis which give complementary results: the best limit on the BR assuming 
a signal discovery with a 5cr significance is obtained with the cut-based analysis, while 
the 95% CL limit obtained with the likelihood-based analysis using the MFL method 
(which takes into account the shape of the discriminant variables) is better. The impact 
of systematic errors on the final results was also studied. The expected branching ratio 
sensitivities obtained by the different analysis and the previous ones [15,18,19] have the 
same order of magnitude, in the range from 10~ 3 to 10~ 5 (for L = 10 fb _1 ). Even if 
the SM predicts a much lower branching ratio for the FCNC decays of the top quark, 
the expected branching ratios obtained in these analysis are several orders of magnitude 
better then present experimental limits. 

The present 95% CL limits and the expected sensitivity at the HERA (ZEUS, L = 
630 pb- 1 ), Tevatron (CDF, run II [27]) and LHC (ATLAS) for BR(t -> 47, qZ) are 
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summarised in Fig I2"2"l 
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Figure 1: Feynman diagrams for the top quark decays considered in this paper: a) SM decay t — > bW ; 
b) FCNC decay t -> Za; c) FCNC decay t 7 g and FCWC decay t 5 g. 
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Figure 2: Distributions for the hadronic mode of a) reconstructed invariant mass of the lepton pairs, ma 
for the best combination and b) reconstructed invariant mass oft — > ££j for the best combination of 11 j. 
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Figure 3: Distributions for the hadronic mode of a) reconstructed invariant mass of the jet pairs, rrijj for 
the best combination and b) reconstructed invariant mass oft—* jjb for the best combination of jjb. 
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e) f) 
Figure 5: The distributions of relevant variables for the t — ► Zq channel are shown after the preselection 
level: a) number of jets; b) number of b- jets; c) number of leptons; d) transverse momentum of the first 
jet; e) transverse momentum of the first lepton andf) missing transverse momentum. The SM background 
is normalised to L = 10 fb^ 1 and the signal has an arbitrary normalization, but the same in all plots of 
this figure. 
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Figure 6: The reconstructed masses after the final selection level for the t — > Zq channel are shown: a) Z 
boson (£ + £~ invariant mass); b) t-quark with FCNC decay (j£ + £~ invariant mass) and c) t-quark with 
SM decay (b£v invariant mass). The SM background is normalised to L = 10 fb^ 1 and the signal has an 
arbitrary normalization, but the same in all plots of this figure. 
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Figure 7: The distribution of the variables based on which the p.d.f. were built are shown (t — > Zq 
channel): a) two leptons minimum mass (only the first three leptons were considered); b) transverse 
momentum of the third lepton and c) transverse momentum of the most energetic non b-jet. The j£ + £~ 
invariant mass was also used as p.d.f. and is shown in Fig. The SM background is normalised to 
L = 10 fb^ 1 and the signal has an arbitrary normalization, but the same in all plots of this figure. 
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Figure 9: The number of expected SM background as a function of the signal efficiency for the t — ► Zq 
channel is shown. The SM background is normalised to L = 10 fb . The arrow shows the point with 
best S/VB. 
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Figure 10: The distributions of relevant variables for the t — > jq channel are shown after the preselection 
level: a) number ofb-jets and b) transverse momentum of the c(u)-jet. The SM background is normalised 
to L = 10 fb -1 and the signal has an arbitrary normalization, but the same in all plots of this figure. 
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Figure 11: The distribution of the variables based on which the p.d.f. were built are shown (t — ► *yq 
channel): a) reconstructed mass of the t-quark with FCNC decay (j'7 invariant mass); b) transverse 
momentum of the photon and c) number of jets. The SM background is normalised to L — 10 fb^ 1 and 
the signal has an arbitrary normalization, but the same in all plots of this figure. 
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Figure 12: SM background and signal discriminant variable distributions for the t ~ * 7(7 channel are 
shown. The SM background is normalised to L = 10 fb^ 1 and the signal has an arbitrary normalization. 
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Figure 13: The number of expected SM background as a function of the signal efficiency for the t — > 
channel is shown. The SM background is normalised to L = 10 fb ■ The arrow shows the point with 
best S/VB. 
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Figure 14: The distributions of relevant variables for the t — > gq ("3 jets") channel are shown after the 
preselection level: a) transverse momentum of the gluon: b) the qg invariant mass and c) the biv invariant 
mass. The SM background is normalised to L = 10 fb^ 1 and the signal has an arbitrary normalization, 
but the same in all plots of this figure. 
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Figure 15: The distribution of the variables based on which the p.d.f. were built are shown (t — > gq 
channel — "3 jets") a) the qg invariant mass; b) the btv invariant mass; c) transverse momentum of the 
b-jet; d) transverse momentum of the c(u)-jet and e) angle between the lepton and the gluon. The SM 
background is normalised to L = 10 fb^ 1 and the signal has an arbitrary normalization, but the same in 
all plots of this figure. 
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Figure 16: Expected background and signal discriminant variable distributions for the t — > gq channel 
with the number of jets equal to three. The SM background is normalised to L — 10 fb^ 1 and the signal 
has an arbitrary normalization. 
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Figure 1 7: The number of expected SM background as a function of the signal efficiency for the t — * gq 
channel with the number of jets equal to three is shown. The SM background is normalised to L = 10 fb^ 1 . 
The arrow shows the point with best S/ \[B. 
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Figure 18: The distributions of relevant variables for the t — > gq ("4 jets") channel are shown after the 
preselection level: a) transverse momentum of the gluon; b) the qg invariant mass and c) the blv invariant 
mass. The SM background is normalised to L — 10 fb^ 1 and the signal has an arbitrary normalization, 
but the same in all plots of this figure. 
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Figure 19: The distribution of the variables based on which the p.d.f. were built are shown (t — > gq channel 
- "4 jets") a) minimum invariant mass of the first and the second non-b jets or the first and the third 
non-b jets; b) the biv invariant mass; c) reconstructed transverse momentum of the qg; d) reconstructed 
transverse momentum of the biv; e) angle between the lepton and the gluon; f) angle between the lepton 
and the b-jet and g) angle between the gluon and the second non-b jet. The SM background is normalised 
to L = 10 fb~ x and the signal has an arbitrary normalization, but the same in all plots of this figure. 
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Figure 20: Expected background and signal discriminant variable distributions for the t — > gq channel 
with the number of jets greater than three. The SM background is normalised to L = 10 fb^ 1 and the 
signal has an arbitrary normalization. 
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Figure 21: The number of expected SM background as a function of the signal efficiency for the t — ► gq 
channel with the number of jets greater than three is shown. The SM background is normalised to L = 



10 fb 1 . The arrow 



shows the point with best S/ \f~B~. 
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Figure 22: The present 95% CL limits on the BR(t — > q^) vs. BR(t — ► qZ) plane are shown. The 
expected sensitivity at the HERA (L = 630 pb^ 1 ), Tevatron (run II) and LHC is also represented. 
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